(Openinference Migration: Langchain): Capture multimodal image content (OpenAI image_url and Anthropic image blocks) as Blob/Uri message parts. - #296
Conversation
Pull request dashboard statusWaiting on the author · refreshed 2026-08-22 17:13 UTC Two things need attention:
Status above doesn't look right?
|
There was a problem hiding this comment.
Pull request overview
Adds multimodal image support to the LangChain GenAI instrumentation by converting OpenAI image_url and Anthropic image blocks into Blob / Uri message parts, and extends the unit test suite to validate the new parsing behavior.
Changes:
- Add multimodal image parsing helpers (
_media_part,_image_from_url) to convert LangChain image blocks intoBlob/Uriparts. - Extend callback-handler tests to cover OpenAI and Anthropic image content shapes (including data URIs and base64 sources).
- Add a changelog fragment documenting the new capability.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| instrumentation/opentelemetry-instrumentation-genai-langchain/src/opentelemetry/instrumentation/genai/langchain/utils.py | Adds image block parsing and base64 decoding to emit Blob/Uri message parts. |
| instrumentation/opentelemetry-instrumentation-genai-langchain/tests/test_callback_handler.py | Adds unit tests for data-URI, HTTP-URI, and Anthropic image/source parsing into message parts. |
| instrumentation/opentelemetry-instrumentation-genai-langchain/.changelog/296.added | Documents the new multimodal image capture support. |
aa26c4f to
ed3949d
Compare
d72b64e to
501725d
Compare
|
@rads-1996 One thing I wanted to check on ordering. In That means a config that doesn't record content still copies large images. If that isn't intended, either skip the media decode when capture is off, or add a test asserting If the parse is meant to run unconditionally so the completion hook has something to work with, that's fine, I just wanted to confirm that's the assumption. |
@AgentGymLeader Thanks for the callout. I have gated decoding and will skipped when the content capture mode is |
9cd26fe to
1b98bc4
Compare
aaea6b5 to
d9fe119
Compare
9ae1c71 to
0c19a81
Compare
eb9e373 to
c57a7b7
Compare
f0b8684 to
837f846
Compare
3f3164a to
e21d1ab
Compare
|
Hi @rads-1996 — just a friendly reminder that this pull request is waiting on you. The dashboard status comment has the open items and is kept current.
|
72b07c3 to
7e4edc4
Compare
…mage` blocks) as `Blob`/`Uri` message parts.
9f7131e to
a0bb771
Compare
| if not url: | ||
| return None | ||
| return image_from_url(url) | ||
| if block_type == "image": |
There was a problem hiding this comment.
LangChain's own standard image blocks are silently dropped - only the raw provider shapes are handled. ImageContentBlock in langchain-core 1.x is {"type": "image", "url"/"base64", "mime_type"}, and 0.3 uses source_type + data; neither has a source key. Since standard blocks are the documented v1 way to pass images, this misses the common case (the whole message is dropped, not just the image part).
This fails on the branch - all four cases:
@pytest.mark.parametrize(
"block,expected",
[
({"type": "image", "base64": B64, "mime_type": "image/png"}, Blob),
({"type": "image", "url": "https://example.com/a.png"}, Uri),
({"type": "image", "source_type": "base64", "data": B64, "mime_type": "image/png"}, Blob),
({"type": "image", "source_type": "url", "url": "https://e/b.png"}, Uri),
],
)
def test_langchain_standard_image_blocks_are_captured(block, expected):
messages = to_input_messages([HumanMessage(content=[block])])
assert messages, "message dropped entirely - no parts extracted"
assert any(isinstance(p, expected) for p in messages[0].parts)Please handle these shapes and add the tests.
| return None | ||
| content = decoded | ||
| else: | ||
| content = payload.encode("utf-8") |
There was a problem hiding this comment.
A non-base64 data: URL is percent-encoded (RFC 2397), so this stores the escaped text rather than the bytes. Use urllib.parse.unquote_to_bytes(payload).
Fails on the branch:
def test_image_from_url_percent_encoded_data_url():
part = image_from_url("data:image/svg+xml,%3Csvg%2F%3E")
assert part.content == b"<svg/>" # got b"%3Csvg%2F%3E"Please fix and add the test.
|
|
||
|
|
||
| def decode_base64(data: str) -> bytes | None: | ||
| """Called only when content capture is enabled |
There was a problem hiding this comment.
nit: docstring of a public function should say what it does - this summary line is a caller precondition.
Description
Part of the langchain migration PRs - #272
Type of change
Please delete options that are not relevant.
How has this been tested?
Please describe the tests that you ran to verify your changes. Provide
instructions so we can reproduce. List any relevant details for your test
configuration.
Checklist
See CONTRIBUTING.md
for the style guide, changelog guidance, and more.